Black-Box Data-efficient Policy Search for Robotics

The most data-efficient algorithms for reinforcement learning (RL) in robotics are based on uncertain dynamical models: after each episode, they first learn a dynamical model of the robot, then they use an optimization algorithm to find a policy that maximizes the expected return given the model and its uncertainties. It is often believed that this optimization can be tractable only if analytical, gradient-based algorithms are used; however, these algorithms require using specific families of reward functions and policies, which greatly limits the flexibility of the overall approach. In this paper, we introduce a novel model-based RL algorithm, called Black-DROPS (Black-box Data-efficient RObot Policy Search) that: (1) does not impose any constraint on the reward function or the policy (they are treated as black-boxes), (2) is as data-efficient as the state-of-the-art algorithm for data-efficient RL in robotics, and (3) is as fast (or faster) than analytical approaches when several cores are available. The key idea is to replace the gradient-based optimization algorithm with a parallel, black-box algorithm that takes into account the model uncertainties. We demonstrate the performance of our new algorithm on two standard control benchmark problems (in simulation) and a low-cost robotic manipulator (with a real robot).Comment: Accepted at the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2017; Code at http://github.com/resibots/blackdrops; Video at http://youtu.be/kTEyYiIFGP

Population dynamics with a mixed type of sexual and asexual reproduction in a fluctuating environment

<p>Abstract</p> <p>Background</p> <p><it>Carassius gibelio</it>, a cyprinid fish from Eurasia, has the ability to reproduce both sexually and asexually. This fish is also known as an invasive species which colonized almost all continental Europe, most likely originating from Asia and Eastern Europe. Populations of both sexually and asexually reproducing individuals exist in sympatry. In this study we try to elucidate the advantages of such a mixed type of reproduction. We investigate the dynamics of two sympatric populations with sexual and asexual reproduction in a periodically fluctuating environment. We define an individual-based computational model in which genotypes are represented by <it>L </it>loci, and the environment is composed of <it>L </it>resources for which the two populations compete.</p> <p>Results</p> <p>Our model demonstrates advantageous population dynamics where the optimal percentage of asexual reproduction depends on selection strength, on the number of selected loci and on the timescale of environmental fluctuations. We show that the sexual reproduction is necessary for "generating" fit genotypes, while the asexual reproduction is suitable for "amplifying" them. The simulations show that the optimal percentage of asexual reproduction increases with the length of the environment stability period and decrease with the strength of the selection and the number of loci.</p> <p>Conclusions</p> <p>In this paper we addressed the advantages of a mixed type of sexual and asexual reproduction in a changing environment and explored the idea that a species that is able to adapt itself to environmental fluctuation can easily colonize a new habitat. Our results could provide a possible explanation for the rapid and efficient invasion of species with a variable ratio of sexual and asexual reproduction such as <it>Carassius gibelio</it>.</p

A Process Calculus for Molecular Interaction Maps

We present the MIM calculus, a modeling formalism with a strong biological basis, which provides biologically-meaningful operators for representing the interaction capabilities of molecular species. The operators of the calculus are inspired by the reaction symbols used in Molecular Interaction Maps (MIMs), a diagrammatic notation used by biologists. Models of the calculus can be easily derived from MIM diagrams, for which an unambiguous and executable interpretation is thus obtained. We give a formal definition of the syntax and semantics of the MIM calculus, and we study properties of the formalism. A case study is also presented to show the use of the calculus for modeling biomolecular networks.Comment: 15 pages; 8 figures; To be published on EPTCS, proceedings of MeCBIC 200

Shell Model for Drag Reduction with Polymer Additive in Homogeneous Turbulence

Recent direct numerical simulations of the FENE-P model of non-Newtonian hydrodynamics revealed that the phenomenon of drag reduction by polymer additives exists (albeit in reduced form) also in homogeneous turbulence. We introduce here a simple shell model for homogeneous viscoelastic flows that recaptures the essential observations of the full simulations. The simplicity of the shell model allows us to offer a transparent explanation of the main observations. It is shown that the mechanism for drag reduction operates mainly on the large scales. Understanding the mechanism allows us to predict how the amount of drag reduction depends of the various parameters in the model. The main conclusion is that drag reduction is not a universal phenomenon, it peaks in a window of parameters like Reynolds number and the relaxation rate of the polymer

The folding of a metallopeptide

We have applied solid-phase synthesis methods for the construction of tris(bipyridyl) peptidic ligands that coordinate Fe(II) ions with high affinity and fold into stable mononuclear metallopeptides. The main factors influencing the folding pathway and chiral control of the peptidic ligands around the metal ions have been studied both by experimental techniques (CD, UV-vis and NMR) and molecular modeling tools. Amongst the numerous molecular variables that have been studied, this study clearly illustrates how the chirality of a given set of aminoacids (proline in this case) of the peptide dictates the chirality of the metal center of the resulting metallopeptide. Moreover, the relatively hydrophobic peptidic models used in this work show that the most stable structures present reduced solvent contacts and, in counterpart, stabilize the cis configuration of the proline residuesWe are thankful for the support given by the Spanish grants SAF2013-41943-R, CTQ2012-31341, CTQ2011-23336 and CTQ2013-49317-EXP; the ERDF and the European Research Council (Advanced Grant 340055); the Xunta de Galicia grants GRC2013-041 and PGIDIT08CSA-047209PR and the Generalitat de Catalunya grant 2009SGR68. Support of COST Action CM1105 is kindly acknowledged. G.R. thanks the INL for his PhD fellowshipS

Black-Box Data-efficient Policy Search for Robotics

International audienceThe most data-efficient algorithms for reinforcement learning (RL) in robotics are based on uncertain dynam-ical models: after each episode, they first learn a dynamical model of the robot, then they use an optimization algorithm to find a policy that maximizes the expected return given the model and its uncertainties. It is often believed that this optimization can be tractable only if analytical, gradient-based algorithms are used; however, these algorithms require using specific families of reward functions and policies, which greatly limits the flexibility of the overall approach. In this paper, we introduce a novel model-based RL algorithm, called Black-DROPS (Black-box Data-efficient RObot Policy Search) that: (1) does not impose any constraint on the reward function or the policy (they are treated as black-boxes), (2) is as data-efficient as the state-of-the-art algorithm for data-efficient RL in robotics, and (3) is as fast (or faster) than analytical approaches when several cores are available. The key idea is to replace the gradient-based optimization algorithm with a parallel, black-box algorithm that takes into account the model uncertainties. We demonstrate the performance of our new algorithm on two standard control benchmark problems (in simulation) and a low-cost robotic manipulator (with a real robot)

Epidemiology and economic impact of moderate and severe neurotrophic keratopathy in Italy

Neurotrophic keratopathy is a rare corneal disease caused by impaired corneal innervation. There is a paucity of published evidence on neurotrophic keratopathy with no published studies on the economics of neurotrophic keratopathy in the Italian or international literature. This cost analysis aimed at assessing the economic impact of moderate (persistent epithelial defect) and severe (corneal ulcer without perforation) neurotrophic keratopathy from the perspective of the National Health Service and patients in Italy. Treatment algorithm and health resource use information were collected from a panel of nine experts from Italian centres specialized in ocular/corneal conditions. National ambulatory and inpatient hospital tariffs were applied to units of service, and Agenzia Italiana del Farmaco (AIFA) published prices to pharmaceuticals. Mean annual per patient cost was derived as an average cost weighted by the proportion of patients on each respective treatment and length of the treatment. The National Health Service + patient perspective additionally included patients' out-of-pocket expenses. The mean annual estimated National Health Service cost of treatment was €5167 (persistent epithelial defect) and €10,885 (corneal ulcer without perforation) per patient. Costs were largely driven by ambulatory visits and hospital interventions. The mean annual estimated National Health Service + patient cost was €5731 (persistent epithelial defect) and €11,478 (corneal ulcer without perforation) per patient, including cost of out-of-pocket expenses for pharmaceuticals and therapeutic contact lenses. Mean annual cost of neurotrophic keratopathy in Italy doubles with disease severity. Further research is warranted to provide more insight especially into societal costs

p38γ/δ activation alters cardiac electrical activity and predisposes to ventricular arrhythmia

We gratefully acknowledge L. Sen-Martín, J. Alegre-Cebollada (CNIC, Madrid) and L. Carrier (University Medical Center HamburgEppendorf and DZHK, Hamburg) for the cMyBP3-C KO cardiac tissue; D. Roiz-Valle and C. López-Otín (IUOPA; Universidad de Oviedo, Oviedo) for the LmnaG609G/G609G cardiac tissue; and R. J. Davis for the MKK6 KO mice. We thank G. Giovinazzo and the CNIC Pluripotent Cell Technology Unit (CNIC, Madrid) for the hiPSCs. We thank S. Bartlett and F. Chanut (CNIC, Madrid) for English editing, and R. R. Mondragon (University of Michigan, Ann Arbor) for technical support. We are grateful to R. J. Davis (University of Massachusetts Chan Medical School, Worcester), A. Padmanabhan (University of California, San Francisco) and M. Costa and C. López-Otín (IUOPA; Universidad de Oviedo, Oviedo) for critical reading of the manuscript. We thank the staf at the CNIC Genomics and Bioinformatics Units for technical support and help with data analysis and A. C. Silva for help with figure editing and design. This work was funded by a CNIC Intramural Project Severo Ochoa (Expediente 12- 2016 IGP) to G.S. and J.J. G.S. is supported by the following projects: PMP21/00057 IMPACT-2021, funded by the Instituto de Salud Carlos III (ISCIII), and PDC2021-121147-I00 and PID2019-104399RB-I00, funded by MCIN/AEI/10.13039/501100011033—all funded by the European Union (FEDER/FSE); ‘Una manera de hacer Europa’/‘El FSE invierte en tu futuro’/Next Generation EU and co-funded by the European Union/Plan de Recuperación, Transformación y Resiliencia (PRTR). R.R.B. is a fellow of the FPU Program (FPU17/03847). B.G.T. was a fellow of the FPI Severo Ochoa CNIC Program (SVP‐2013‐067639) and an American Heart Association Postdoctoral Fellow (18POST34080175). The following grants provided additional funding: Instituto de Salud Carlos III, PDC2021-121147-I00 Convocatoria: Proyectos Prueba de Concepto 2021 Ministerio de Ciencia e Innovación and PID2022-138525OB-I00 Ministerio de Ciencia e Innovación; US National Heart, Lung, and Blood Institute (R01 grant HL122352); Fondos FEDER, Madrid, Spain, and Fundación Bancaria ‘La Caixa (project HR19/52160013) to J.J.; American Heart Association Postdoctoral Fellowship 14POST17820005 to D.P.B.; and MICINN PGC2018-097019-B-I00, ISCIII-SGEFI/ERDF (PRB3-IPT17/0019, ProteoRed), the Fundació Marató TV3 (grant 122/C/2015) and ‘la Caixa’ Banking Foundation (project code HR17- 00247) to J.V. The CNIC is supported by the Instituto de Salud Carlos III (ISCIII), the Ministerio de Ciencia e Innovación (MCIN) and the Pro CNIC Foundation and is a Severo Ochoa Center of Excellence (grant CEX2020-001041-S, funded by MICIN/AEI/10.13039/501100011033).S